165,822 research outputs found
Exploring efficient neural architectures for linguistic-acoustic mapping in text-to-speech
Conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models such as recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure with intermediate affine transformations tends to make them slow to train and to sample from. In this work, we explore two different mechanisms that enhance the operational efficiency of recurrent neural networks, and study their performance–speed trade-off. The first mechanism is based on the quasi-recurrent neural network, where expensive affine transformations are removed from temporal connections and placed only on feed-forward computational directions. The second mechanism includes a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder networks are competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU and GPU inference time. The best performing model is the one based on the quasi-recurrent mechanism, reaching the same level of naturalness as the recurrent neural network based model with a speedup of 11.2 on CPU and 3.3 on GPU.Peer ReviewedPostprint (published version
Learning text representation using recurrent convolutional neural network with highway layers
Recently, the rapid development of word embedding and neural networks has
brought new inspiration to various NLP and IR tasks. In this paper, we describe
a staged hybrid model combining Recurrent Convolutional Neural Networks (RCNN)
with highway layers. The highway network module is incorporated in the middle
takes the output of the bi-directional Recurrent Neural Network (Bi-RNN) module
in the first stage and provides the Convolutional Neural Network (CNN) module
in the last stage with the input. The experiment shows that our model
outperforms common neural network models (CNN, RNN, Bi-RNN) on a sentiment
analysis task. Besides, the analysis of how sequence length influences the RCNN
with highway layers shows that our model could learn good representation for
the long text.Comment: Neu-IR '16 SIGIR Workshop on Neural Information Retrieva
Equivalence of Equilibrium Propagation and Recurrent Backpropagation
Recurrent Backpropagation and Equilibrium Propagation are supervised learning
algorithms for fixed point recurrent neural networks which differ in their
second phase. In the first phase, both algorithms converge to a fixed point
which corresponds to the configuration where the prediction is made. In the
second phase, Equilibrium Propagation relaxes to another nearby fixed point
corresponding to smaller prediction error, whereas Recurrent Backpropagation
uses a side network to compute error derivatives iteratively. In this work we
establish a close connection between these two algorithms. We show that, at
every moment in the second phase, the temporal derivatives of the neural
activities in Equilibrium Propagation are equal to the error derivatives
computed iteratively by Recurrent Backpropagation in the side network. This
work shows that it is not required to have a side network for the computation
of error derivatives, and supports the hypothesis that, in biological neural
networks, temporal derivatives of neural activities may code for error signals
- …